Search AI Products and News

AI security company Adversa AI has released a shocking report stating that Elon Musk's startup xAI's newly launched Grok3 model has significant vulnerabilities in terms of cybersecurity. Adversa's research team found that this latest AI model is susceptible to 'simple jailbreak attacks', potentially allowing malicious actors to access sensitive information such as 'how to groom children, handle corpses, extract DMT, and make bombs'. Worse still, Adversa

Within just six days, participants successfully bypassed all security measures of Anthropic's AI model, Claude3.5, sparking new discussions in the field of AI security. Jan Leike, a former member of OpenAI's alignment team now at Anthropic, announced on the X platform that one participant managed to breach all eight security levels. This collective effort involved approximately 3,700 hours of testing and 300,000 messages from participants. Despite the challengers

Recent research indicates that the most advanced AI chatbots on the market are surprisingly sensitive to simple tricks, making them easy to 'jailbreak.' According to a report by 404 Media, the developers of the Claude chatbot, Anthropic, found that by intentionally introducing spelling errors into the prompts, these large language models can be led to ignore their own safety mechanisms and generate content they are supposed to refuse. Image source note: Image generated by AI, image licensed by service provider Midjourney.

Recently, researchers from Stanford University and the University of Hong Kong discovered that current AI Agents (such as Claude) are more easily disrupted by pop-ups than humans, with significantly reduced performance even when faced with simple pop-ups. According to the study, AI Agents achieved an average attack success rate of 86% when confronted with deliberately designed pop-ups in experimental environments, leading to a 47% decrease in task success rate. This finding has sparked new concerns regarding the safety of AI Agents, especially as they are assigned more autonomous tasks.

Anthropic has announced an expansion of its vulnerability reward program aimed at testing a 'next-generation AI safety mitigation system,' primarily focusing on identifying and defending against 'universal jailbreak attacks.' Special attention is given to high-risk areas, including CBRN defense and cybersecurity. Participants will have the opportunity to engage with the new safety system ahead of time, identifying vulnerabilities or bypassing security measures, with rewards of up to $15,000. This initiative aims to enhance the safety of AI systems by attracting security researchers to collaboratively discover and fix potential threats, setting a benchmark for safety in the AI industry.

Deep learning-based software security detection systems act as the 'security inspectors' of the digital age, efficiently identifying software vulnerabilities. However, a study named EaTVul has unveiled new challenges in this field. EaTVul is an innovative evasion attack strategy that modifies vulnerable code, leading deep learning detection systems to misjudge, achieving success rates ranging from 83% to 100%. Its method includes using support vector machines to identify key samples, attention mechanisms to recognize critical features, AI chatbots to generate confusing data, and fuzzy logic.

The Prompt-Guard-86M model released by Meta is designed to defend against prompt injection attacks by restricting large language models from processing inappropriate inputs, thereby protecting system security. However, the model itself also exposes risks of being attacked. Research conducted by Aman Priyanshu found that by adding simple character spacing such as spaces or removing punctuation in the input, the model disregards prior security instructions, achieving an almost 100% success rate for attacks. This finding highlights the importance of AI security, despite Prompt

A joint research team from Beihang University and Nanyang Technological University conducted extensive security tests on the GPT-4o model. Through thousands of API queries, researchers evaluated the security of text, image, and audio modalities of GPT-4o. The study found that while GPT-4o showed improvements in security against text jailbreak attacks, the newly introduced audio modality presented new attack surfaces, and its overall multimodal security was not as strong as the previous generation model GPT-4V.

["OpenAI claims that The New York Times hired hackers to attack ChatGPT","OpenAI argues that the hiring of hackers by The New York Times is critical","The New York Times used prompts to attack ChatGPT","OpenAI retaliates against blatant violations of product usage terms","AI companies face a copyright lawsuit crisis similar to Google's","Internet companies selling data to AI companies","Proposal for the latest data selection research framework","AI-generated content accelerates its spread across the Internet","Large models admit to using AI-generated data"]

Teams from Google DeepMind, the University of Washington, and others have successfully unveiled the sources of ChatGPT's training data through a 'poetry attack'. This attack method involves repeating specific vocabulary, which led ChatGPT to disclose personal information, including phone numbers and email addresses. The vulnerability was reported to OpenAI on August 30, and no further comments have been made. The research calls for environmentally-friendly AI usage, pointing out the significant energy burden that text and image generation imposes on the environment.

OpenAI's DALL-E 3, an image-generating AI technology, has been found to be easily influenced by jailbreak prompts, resulting in the generation of photos depicting children smoking. The jailbreak technique was discovered by LinkedIn user Peter Gostev, who shared how to successfully bypass the AI's restrictions to create images related to children smoking. Although the jailbreak prompts are quite complex, this incident serves as a reminder that even the wealthiest companies in Silicon Valley struggle to establish comprehensive protective measures for their AI systems.

Foreign netizens have discovered a new jailbreak technique that allows ChatGPT to generate ransomware using disordered prompts. By taking advantage of the human brain's ability to understand scrambled words and phrases, they bypass security filters. Jim Fan was amazed at the GPT model's comprehension of disordered words.

This article introduces a new type of cyber attack where hackers can steal personal data by listening to the sounds of keyboard inputs. This attack utilizes a custom deep learning-based algorithm to perform acoustic analysis of keyboard sounds and automatically decode the input content. Research shows that it can accurately decode 95% of the input content through this method. The article explains the principle of this attack and possible defense measures.